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(54) Implementing a scalable, dynamic, fault-tolerant, multicast based file transfer and 
asynchronous file replication protocol 



(57) Apparatus and method to improve the speed, 
scalability, robustness and dynamism of multicast data 
transfers to remote computers. Many Grid Computing 
applications, such as Genomics, Proteomics, Seismic, 
Risk Management, etc, require a priori transfer of sets 
of files or other data to remote computers prior to 
processing taking place. Existing multicast and data 
transfer protocols are static and can not guarantee that 
all nodes will contain a copy of the replicated data or 



files. The fully distributed data transfer and data replica- 
tion protocol of the invention permits transfers which 
minimize processing requirements on master transfer 
nodes by spreading work across the network. The result 
is higher scalability than current centralized protocols, 
more dynamism and allows fault-tolerance by distribu- 
tion of functionality. The ability to distribute the protocol 
is simplified through our innovative symmetric-connec- 
tionless data transfer protocol. 



Figure 1: Symmetric-Connectionless File Transfer Protocol Primitive 
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Description 

FIELD OF THE INVENTION : 

[0001] The present invention relates to transferring 
and replicating data among geographically separated 
computing devices, and, in particular, to implementing 
a multicast file transfer protocol to transfer files more 
rapidly, robustly and to more computing devices than 
current methods permit. In addition, the invention can 
be used to asynchronously maintain a set of replicated 
files throughout computer failures and introduction of 
new computers into the network. 

BACKGROUND OF THE INVENTION : 

[0002] Grid Computers, Computer Farms and similar 
Computer Clusters are being used to deploy a novel 
type of parallel applications based on concurrent inde- 
pendent tasks related only by their individual contribu- 
tion to a global problem's resolution. Until now all paral- 
lel applications were based on splitting a single task into 
a multitude of collaborating subtasks (ie Open MP, PVM, 
MPI, etc). However, in some application areas users 
have recently started to split single large problems into 
a multitude of sub problems which can be resolved in- 
dependently of one another. This methodology allows 
higher scalability and permits the use of Grid Computing 
techniques and the use of cost efficient computing so- 
lutions (ie clusters), but requires that the necessary data 
files first be replicated to the remote nodes prior to the 
computation taking place. It is this problem of replicated 
data transfers that our invention addresses. 
[0003] Existing art to address data file transfer falls 
into three categories. 

[0004] First, tasks can make use of on-demand file 
transfer apparatus, better known as file servers. For 
problems where file access is minimal, this type of so- 
lution works as long as the cluster size (ie number of 
remote computers) is limited to a few hundred. For large 
and frequent file accesses, this solution does not scale 
beyond a handful of nodes. Moreover, if entire data files 
are accessed by all nodes, the total amount of data 
transfer will be N times that of a single file transfer 
(where N is the number of nodes). This waste of network 
bandwidth limits scalability and penalizes computational 
performance as the nodes are blocked waiting for re- 
mote data. 

[0005] Second, users or tasks can manually transfer 
files prior to execution though a point-to-point file trans- 
fer protocol. There are three types of point-to-point pro- 
tocols. Standard file transfer protocols (ie ftp, tftp) where 
one file is transferred to one remote node, one packet 
at a time. Sliding window file transfer protocols, such as 
the "parallel file transfer protocol" from Donald J. Faboz- 
zi II where multiple packets transit concurrently on their 
way to a single remote node. And parallel file transfer 
protocols (ex HPSS PFTP) where multiple point-to-point 



file transfers operate concurrently. While these methods 
improve network bandwidth utilization over demand 
based schemes, the final result is the same: a file is 
transferred "N" times over the network when replicating 

5 information unto "N" remote computers. Moreover, ad- 
ditional file transfers must continually be initiated to 
cope with the constantly varying nature of large compu- 
ter networks (ie new nodes being added to increase a 
cluster or Grid size or to replace failed or obsolete 

10 nodes). 

Third, users or tasks can manually transfer file prior to 
execution through a multicast (or broadcast) file transfer 
protocol (ex StarBurst SMFTP). In this scheme each file 
fragment sent over the network is simultaneously read 

15 by all participating remote computers. Hence network 
bandwidth usage is limited to the same amount of data 
traffic as for a single point-to-point file transfer. This is 
currently the most frequent scheme used to resolve 
problems having been split into multiple concurrent in- 

20 dependent tasks as described above. However, this 
form of apparatus is imperfect. For instance, error re- 
covery is concurrent to the multicast phase. This impos- 
es an increased workload on the master file server node 
and eventually will limit scalablity. These schemes also 

25 are based on the notion of node registration, where prior 
to the multicast phase, all active and participating re- 
mote computers must register to participate in a transfer 
request. Hence, new nodes being booted during or after 
the multicast transfer phase will not be participating in 

30 the effort to replicate files. Another drawback is that reg- 
istered computers which crash during the multicast 
phase can not join back the transfer group after reboot. 
Finally, these schemes can not survive through a crash 
on the master file server (ie the computer which per- 

35 forms the multicast file transfer). These sum of these 
limitations is that current multicast file transfer art work 
fail at their task of insuring correct file replication among 
all participating remote computers in a normal setup of 
dynamic and error prone network of computers. They 

40 lack the fault-tolerance, ability to handle dynamic regis- 
tration, scalability to tens of thousands of nodes and ca- 
pability to persist with the file replication effort once the 
master transfer process terminates. 

45 SUMMARY OF THE INVENTION : 

[0006] The object of the present invention is to imple- 
ment a multicast data transfer apparatus which keeps 
operating through computer failures, allows data repli- 
50 cation scalability to very large size networks, and, which 
persists transferring data to newly introduced nodes 
even after the master data transfer process has termi- 
nated. 

[0007] The terms "computer" and "node" used in the 
55 description of the present invention must be understood 
in the broadest sense, as they can include any comput- 
ing device or any electronic appliance including a com- 
puting device, such as for example a personal computer, 
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a cellular phone, a PDA, etc., which is or can be con- 
nected to any one type of network. 
[0008] The term data transfer used in the description 
of the present invention must be understood in the 
broadest sense, as it can include full and partial data 5 
transfers. That is, it relates to transfers where an entire 
data entity (e.g. file) is transferred at once, as well as 
situations where selected segments of a data entity are 
transferred at some point. An example of the latter case 
is a data entity being transferred in its entirety and at a 10 
later time, selected segments of the data entity are being 
updated. 

[0009] Briefly stated, the present invention ensures 
the correct replication of sets of files or any other data, 
for instance in a network of computers, in spite of net- 15 
work failures, computer crashes, or the introduction of 
new computers in the network. 

[0010] The present invention innovates in the follow- 
ing areas: 

20 

1 . symmetric-connectionless data transfer protocol 
allowing stateless data transfers (ie no need for a 
centralized master data transfer engine to maintain 
individual state information about participating 
nodes); 25 

2. separation of the multicast data transfer phase 
and the point-to-point error recovery phase per- 
formed by two independent protocol engines; 

3. distributed data transfer protocol where all par- 
ticipating remote computers can collaborate in the 30 
error recovery and data replication phases; 

4. use of the recovery phase protocol to enable 
crashed computers to complete data transfers upon 
reboot; 

5. use of the recovery phase protocol to enable 35 
newly introduced nodes to perform asynchronously 
recent data transfers having occurred before they 
became operational (ie data replication); 

6. automatic removal of replicated data once they 
reach their pre-set life span; 40 

7. fault-tolerance of the master data transfer proc- 
ess; 

8. dynamically adaptable peer process selection 
mechanism through a random number and modulus 
calculation scheme; 45 

9. full and partial (ie segments of files) datatransfers 
are supported through the same apparatus. 

[001 1 ] The apparatus and method according to the in- 
vention improve the speed, scalability, robustness and 50 
dynamism of multicast data transfers to remote comput- 
ers. Many Grid Computing applications, such as Ge- 
nomics, Proteomics, Seismic, Risk Management, etc, 
require a priori transfer of sets of files or other data to 
remote computers prior to processing taking place. Ex- 55 
isting multicast and data transfer protocols are static and 
can not guarantee that all nodes will contain a copy of 
the replicated data or files. The fully distributed data 



transfer and data replication protocol of the invention 
permits transfers which minimize processing require- 
ments on master transfer nodes by spreading work 
across the network. The result is higher scalability than 
current centralized protocols, more dynamism and al- 
lows fault-tolerance by distribution of functionality. The 
ability to distribute the protocol is simplified through our 
innovative symmetric-connectionless data transfer pro- 
tocol. 

[001 2] In particular, the present invention is preferably 
embodied by a method to using a file transfer protocol 
to transfer, without regards to a user's privilege, files be- 
tween remote computers, comprising: 

1 . segmenting a file into a number of data packets 
to be multicast (or broadcasted) over a network of 
computers; 

2. recording in a log at each receiving computer the 
segments of the transferred file already received 
and those still missing; 

3. rebuilding the transferred file by writing received 
data packets at their original respective location in 
the file using direct access IO; 

4. transmitting by a multicast, or broadcast, appa- 
ratus said packets over a network of computers; 

5. recovering of missing, incomplete or corrupted 
data packets by means of a distributed transfer re- 
covery apparatus independent from the transfer ap- 
paratus used initially to multicast the data packets; 

6. completing of interrupted file transfers by cause 
of node failure upon reboot by means the recovery 
apparatus; 

7. pursuing file transfers in spite of root transfer 
node failure by the automatic selection of an alter- 
nate multicast root transfer node; 

8. synchronizing replicated files upon reboot, or the 
addition in the network, of a node by means of the 
recovery apparatus; 

9. removing partially transferred files on the remote 
nodes upon canceling or aborting the file transfer 
request by the user, an operator or a system crash 
of the requesting node; 

10. determining the number of operational nodes 
which are in the process of completing an in- 
prog ress file transfer or have already completed a 
file transfer; 

11. removing automatically replicated files which 
have exceeded their preset life-span; 

12. selecting peer processes (transfer master se- 
lection, transfer error recovery and file replication) 
through a dynamically adaptable random number 
and modulus calculation scheme. 

BRIEF DESCRIPTION OF THE DRAWINGS : 

[0013] 

Figure 1 illustrates the symmetric-connectionless 
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file transfer protocol primitive; 
Figure 2 illustrates the layout of the broadcast/mul- 
ticast file transfer process; 

Figure 3 depicts the layout of the transfer error re- 
covery and file replication process; 
Figure 4 shows the user interface process protocol 
finite state machine; 

Figure 5 shows the file transfer master process pro- 
tocol finite state machine; 

Figure 6 illustrates the file transfer slave process 
protocol finite state machine; 
Figure 7 shows the multicast/broadcast master 
process protocol finite state machine; 
Figure 8 depicts the forwarder slave process proto- 
col finite state machine; 

Figure 9 shows the transfer error recovery slave 
process protocol finite state machine; 
Figure 10 illustrates the file replication slave proc- 
ess protocol finite state machine; 
Figure 11 shows the distributed selection mecha- 
nism. 

DETAILED DESCRIPTION OF THE INVENTION : 

[0014] Figure 1 summarizes the protocol primitive 
used to implement the symmetric-connectionless file 
transfer protocol. This protocol primitive is said to be 
connectionless (ie redundant) because it contains all in- 
formation required to perform a file transfer in every data 
packet exchange. Indeed file name, file flags, life span, 
file size, etc are duplicated in each data packet. This 
information redundancy consumes less than 5% of 
packet space (Ethernet MTU of 1500bytes), but allows 
remote computers to easily "jump in" to any file transfer 
multicast phase without prior registration phase. More- 
over, it allows simple and efficient error recovery and file 
synchronization for newly introduced nodes and out-of- 
order processing of data packets. The data transfer 
primitive is further said to be symmetric because it can 
be used by the master file transfer process (during the 
multicast transfer phase) or by any other participating 
nodes (for error recovery or file replication purposes). 
[0015] Figure 2 shows the different processes layout 
to complete a multicast file transfer. A user interface 
process is launched by a user or automated tool to reach 
all active file transfer master processes and initiate the 
multicast file transfer. The scope of interaction between 
these two process types is defined by the geographic 
coverage of the first multicast/broadcast group. One file 
transfer master process is selected to proceed to the 
actual multicast to all active file transfer slave processes 
reachable by in the second multicast/broadcast group. 
[0016] Figure 3 depicts the types of peer-to-peer (ie 
symmetric) exchanges among file transfer slave proc- 
esses during a file transfer error recovery phase or a file 
replication phase. The geographic scope is delimited by 
the second multicast/broadcast group coverage. 
[0017] Figures 4 through 10 show the finite state ma- 



chines used to implement the multicast/broadcast file 
transfer and file replication protocols for the user inter- 
face, file transfer master and file transfer slave process- 
es and their related sub-processes. The mode of oper- 

5 ation can allow multiple concurrent multicast/broadcast 
file transfers and overlapping of multicast/broadcast file 
transfer, transfer error recovery and file replication phas- 
es. Fault-tolerance, scalability and dynamism are 
achieved through real-time peer selection and commu- 

10 nication persistence. 

[0018] Referring to Figure 1 , all preceding File Trans- 
fer Protocol art is based on the notion of client-server 
connections or registrations. This requirement prevents 
dynamic client participation in file transfer activities. It 

15 further enforces strict delivery packet ordering. Finally, 
it necessitates a complex reconnection mechanism. Our 
file transfer protocol is, by opposition, based on a con- 
nectionless model where, without any preceding proto- 
col exchange, file fragments can be exchanged among 

20 cooperating processes. Hence at the receiving end 
processes can jump into any ongoing file transfer ex- 
change at any moment in time, and count on the transfer 
error recovery protocol to retrieve earlier packets, or 
missing packets alike. Furthermore, by splitting multi- 

25 cast transfer and recovery transfer phases, connection- 
less data exchanges allow any cooperating process to 
participate in error recovery and file replication, thus the 
symmetric nature of our apparatus. Symmetry also in- 
herently implies higher scalability, since any number of 

30 processes may contribute to the recovery phase (the 
bottleneck of preceding point-to-point recovery arts), 
and fault-tolerance. Finally, a symmetric protocol allows 
asynchronous activities, past the normal termination of 
the multicast file transfer phase. This feature allows the 

35 implementation of a file replication mechanism where 
newly added or rebooted nodes may contact cooperat- 
ing processes to synchronize with past file transfer ac- 
tivities. 

[0019] Figure 2 represents the interconnection of 

40 processes in our apparatus. There are three process 
level components: the user interface, the file transfer 
master and the file transfer slave processes. 
[0020] The user interface is mandated with establish- 
ing, and maintaining established, a link with any one of 

45 the active file transfer masters and sending the file frag- 
ments. The link is established by multicasting (or broad- 
casting) a request on a predefined communication port 
(socket port number) and selecting one of the active file 
transfer master. The presence of multiple file transfer 

50 masters and our communication protocol allows fault- 
tolerance, that is, the multicast file transfer will continue 
through file transfer master processes failures as long 
as there is still at least one active file transfer master. 
Moreover redundant file transfer master allows for con- 

55 current multicast file transfers. A serialization or token 
mechanism may be added to prevent network saturation 
by limiting the number of simultaneous file transfers. 
[0021] Once a file transfer master is selected to per- 
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form the multicast file transfer, it forks a child process to 
take over the multicast (or broadcast) transfer phase, 
allowing a single file transfer master to handle multiple 
transfer requests simultaneously. The child process 
then forwards all file fragments over the network to pre- 
determined communication port for the benefit of all par- 
ticipating file transfer slave processes. Active file trans- 
fer slaves pick up the file fragments from the network 
and write them at their appropriate location in the target 
replicated file. 

[0022] Figure 3 shows the sort of activities, among file 
transfer slave processes, which may persist after the 
multicast transfer phase has terminated. For instance, 
cooperating file transfer slaves may assist each other in 
an error recovery phase, forwarding file fragments to 
other slaves having missed some file fragments or re- 
ceived corrupted file fragments. A simple extension of 
this error recovery protocol allows for newly introduced 
nodes, running afile transfer slave, to catch up on earlier 
file transfers and (re)build their set of locally replicated 
files. 

[0023] The selection mechanism, Figure 11, used by 
a user interface process to elect a file transfer master 
or by a file transfer slave process to choose another file 
transfer slave process to perform file replication or error 
recovery is based on a novel random number and mod- 
ulus calculation scheme. Prior distributed computing 
methods to perform election are based on NxN mes- 
sage exchanges. This NxN problem resolution creates 
network communication bottlenecks in large networks 
with many elections to process and physically prevents 
scaling to tens of thousands of nodes. Moreover it re- 
quires an a priori knowledge of the network topology and 
number of participants. In our scheme, a partner selec- 
tion, among a large set of cooperating candidates, is 
performed by performing a multicast (or broadcast) of a 
random number and a modulus number. Upon recep- 
tion, likely candidates calculate two new random num- 
bers. The first random number is applied the received 
modulus number and if the result matches the received 
random number, the second generated random number 
is sent back. The election originator accumulates re- 
turned answers for a limited amount of time and selects 
the candidate with the smaller returning random 
number. This scheme is made adaptative by varying the 
modulus number in order to reduce or increase the 
number of respondents. The modulus number to use for 
a new election round is based on the number or re- 
spondents from past requests, and initially is set to "1" 
(forces everybody to respond). 

[0024] Figure 4 depicts the user interface process 
protocol finite state machine. The initial step is to select 
a file transfer master process to send file fragments to. 
This phase fails if no file transfer master processes are 
reachable. The transfer of file fragments begins and pro- 
ceeds until all fragments have been transferred. Should 
the selected file transfer master process stop respond- 
ing, a new election round is initiated and transfer may 



proceed from where it was interrupted. 
[0025] The file transfer master process protocol finite 
state machine, Figure 5, is quite minimal; it replies to 
selection requests and, once selected, spawns a child 

5 process to conduct the actual multicast (or broadcast) 
file transfer phase. The multicast (or broadcast) file 
transfer process protocol finite state machine (Figure 7) 
consists in forwarding all file fragments received from 
the user interface process to all participating file transfer 

10 slave processes. Should the user interface process stop 
responding, the multicast file transfer process notifies 
all file transfer slave processes and terminates. The pro- 
tocol may be extended to perform a file transfer comple- 
tion check with all remote file transfer slave processes. 

15 [0026] The file transfer slave process protocol finite 
state machine shown in Figure 6 implements the multi- 
cast file transfer reception side, the transfer error recov- 
ery mechanism and further contains two optional proto- 
col extensions for file transfer completion and file repli- 

20 cation. Single message exchange requests, such as 
completion check, transfer abort request, file replication 
or error recovery selection requests and reception of file 
fragments are handled directly by the slave process. All 
other tasks, such as assisting another slave to recover 

25 file fragments, or initiating a recovery procedure or file 
replication upon boot are handled in individual sub-proc- 
esses. Consequently, asingle file transfer slave process 
can handle multiple simultaneous file transfers and file 
transfer recovery procedures or can assist concurrently 

30 more than one slave process to recover missing or cor- 
rupted file fragments. The optional protocol extensions 
are file completion check and file replication procedure. 
[0027] The file transfer forwarding process, Figure 8, 
consists in forwarding requested file fragments to the 

35 originating file transfer slave process until no further re- 
quests are received during a preset period of time. 
[0028] Figure 9 shows the file transfer recovery proc- 
ess protocol finite state machine. After an initial selec- 
tion phase, to locate a cooperating file transfer slave 

40 process, requests to forward missing (or corrupted) file 
fragments are sent out to the selected slave process. 
Cooperating processes respond to a forwarding request 
only if they possess a proper copy of the file fragment 
requested. If no cooperating slave process can be se- 

45 lected (ie no other slave process contains the requested 
file fragment) the incomplete file is removed and the re- 
covery terminates. Forwarded file fragments, once re- 
ceived, are written in their correct location in the target 
file. 

50 [0029] The overall multicast file transfer and recovery 
mechanism described so far can be further extended to 
perform automatic file replication as depicted in Figure 
10 (file replication process protocol finite state ma- 
chine). Upon startup a file transfer slave process can 

55 spawn a sub-process to perform asynchronously the file 
replication procedure. File replication serves two pur- 
poses: complete upon boot interrupted file transfers and 
perform file transfers that have occurred while the file 
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transfer slave process was non operational. The proto- 
col starts by initiating a selection procedure to locate a 
cooperating file transfer slave process. This cooperating 
process serves the purpose to determine which file 
transfers occurred while the requesting slave process 
was non operational. Afterwards each file transfer 
missed, or interrupted (these can be determined locally 
from the file fragments stored) is completed using the 
normal file recovery protocol engine (either in an inde- 
pendent sub-process or not, depending on the imple- 
mentation). 

[0030] The combination of persistent connectionless 
requests and distributed selection procedure allows for 
scalability and fault-tolerance since there is no need for 
global state knowledge to be maintained by a central- 
ized entity (or replicated entities). Furthermore it allows 
to build a light weight protocol which can be implement- 
ed efficiently even on appliance type devices. The use 
of multicast (or broadcast) minimizes network utilization, 
allowing higher aggregate file transfer rates and ena- 
bling the use of lesser expensive networking equipment 
(which in turn allows the use of lesser expensive nodes). 
The separation of multicast file transfer and recovery file 
transfer phases allows the deployment of a distributed 
file recovery mechanism that further enhances scalabil- 
ity and fault-tolerance properties. Finally, the independ- 
ent file transfer recovery mechanism can be used to im- 
plement an asynchronous file replication apparatus, 
where newly introduced nodes (or rebooted nodes) can 
perform file transfers which occurred while they were 
non operational and after the completion of the multicast 
file transfer phase. 

[0031] In its preferred embodiment, the present inven- 
tion is applied to file transfer and file replication. The one 
skilled in the art will however recognize that the present 
invention can be applied to the transfer, replication and/ 
or streaming of any type of data. 

Claims 

1 . Method to transferring data between computing de- 
vices comprising a data transfer phase using a mul- 
ticast and/or broadcast transfer protocol and further 
comprising an error recovery phase for recovering 
corrupted or missing data, characterized in that 
corrupted or missing data can be recovered from 
any one of said computing devices. 

2. Method according to the preceding claim, said 
transfer protocol being connectionless. 

3. Method according to one of the preceding claims, 
wherein said recovery phase is performed inde- 
pendently from said transfer phase. 

4. Method according to one of the preceding claims, 
said recovery phase being used for transferring al- 



ready transferred data from one of said computing 
devices to a newly connected computing device. 

5. Method according to one of the preceding claims, 
5 said recovery phase being used for completing in- 
terrupted data transfers. 

6. Method according to one of the preceding claims, 
said data being segments of a file. 

10 

7. Method according to one of the preceding claims, 
further comprising recording the received data in a 
log at each computing device of said computing de- 
vices receiving data. 

15 

8. Computing device adapted to performing the meth- 
od of one of the preceding claims. 

9. Computer program product, directly loadable into 
20 the internal memory of a computing device, com- 
prising software code portions for performing the 
method of one of the claims 1 to 7. 
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Figure 1: Symmetric-Connectionless File Transfer Protocol Primitive 
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Figure 2: Broadcast/Multicast File Transfer Process Layout 
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Figure 3: Transfer Error Recovery and File Replication Process Layout 
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Figure 4: User Interface Process Protocol Finite State Machine 



BOOT 




Timeout 
(lost conn) 




Timeout 
0 answers 



Timeout 
>= 1 answers 



■►EXIT 

(no conn) 



Rcvd 

ack! 



Send User Data 
+ 

Reset Timeout 



EOF 



EXIT 
(QK) 



10 



EP 1 365 538 A1 



Figure 5: File Transfer Master Process Protocol Finite State Machine 
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Figure 6: File Transfer Slave Process Protocol Finite State Machine 




Forwarder Process 




optional 




12 



EP 1 365 538 A1 



Figure 7: Multicast/Broadcast Master Process Protocol Finite State Machine 
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Figure 8: Forwarder Slave Process Protocol Finite State Machine 
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Figure 9: Transfer Error Recovery Slave Process Protocol Finite State Machine 
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Figure 10: File Replication Slave Process Protocol Finite State Machine 
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Figure 11: Distributed Selection Mechanism 
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